Convolutional Patch Networks with Spatial Prior for Road Detection and Urban Scene Understanding

Classifying single image patches is important in many different applications, such as road detection or scene understanding. In this paper, we present convolutional patch networks, which are convolutional networks learned to distinguish different image patches and which can be used for pixel-wise labeling. We also show how to incorporate spatial information of the patch as an input to the network, which allows for learning spatial priors for certain categories jointly with an appearance model. In particular, we focus on road detection and urban scene understanding, two application areas where we are able to achieve state-of-the-art results on the KITTI as well as on the LabelMeFacade dataset. Furthermore, our paper offers a guideline for people working in the area and desperately wandering through all the painstaking details that render training CNs on image patches extremely difficult.Comment: VISAPP 2015 pape

Let's Get the FACS Straight -- Reconstructing Obstructed Facial Features

The human face is one of the most crucial parts in interhuman communication. Even when parts of the face are hidden or obstructed the underlying facial movements can be understood. Machine learning approaches often fail in that regard due to the complexity of the facial structures. To alleviate this problem a common approach is to fine-tune a model for such a specific application. However, this is computational intensive and might have to be repeated for each desired analysis task. In this paper, we propose to reconstruct obstructed facial parts to avoid the task of repeated fine-tuning. As a result, existing facial analysis methods can be used without further changes with respect to the data. In our approach, the restoration of facial features is interpreted as a style transfer task between different recording setups. By using the CycleGAN architecture the requirement of matched pairs, which is often hard to fullfill, can be eliminated. To proof the viability of our approach, we compare our reconstructions with real unobstructed recordings. We created a novel data set in which 36 test subjects were recorded both with and without 62 surface electromyography sensors attached to their faces. In our evaluation, we feature typical facial analysis tasks, like the computation of Facial Action Units and the detection of emotions. To further assess the quality of the restoration, we also compare perceptional distances. We can show, that scores similar to the videos without obstructing sensors can be achieved.Comment: VISAPP 2023 pape

Automatically Estimating Forestal Characteristics in 3D Point Clouds using Deep Learning

Biodiversity changes can be monitored using georeferenced and multitempo-ral data. Those changes refer to the process of automatically identifying differ-ences in the measurements computed over time. The height and the Diameterat Breast Height of the trees can be measured at different times. The mea-surements of individual trees can be tracked over the time resulting in growthrates, tree survival, among other possibles applications. We propose a deeplearning-based framework for semantic segmentation, which can manage largepoint clouds of forest areas with high spatial resolution. Our method divides apoint cloud into geometrically homogeneous segments. Then, a global feature isobtained from each segment, applying a deep learning network called PointNet.Finally, the local information of the adjacent segments is included through anadditional sub-network which applies edge convolutions. We successfully trainand test in a data set which covers an area with multiple trees. Two addi-tional forest areas were also tested. The semantic segmentation accuracy wastested using F1-score for four semantic classes:leaves(F1 = 0.908),terrain(F1 = 0.921),trunk(F1 = 0.848) anddead wood(F1 = 0.835). Furthermore,we show how our framework can be extended to deal with forest measurementssuch as measuring the height of the trees and the DBH

Predicting carbon dioxide and energy fluxes across global FLUXNET sites with regression algorithms

Spatio-temporal fields of land–atmosphere fluxes derived from data-driven models can complement simulations by process-based land surface models. While a number of strategies for empirical models with eddy-covariance flux data have been applied, a systematic intercomparison of these methods has been missing so far. In this study, we performed a cross-validation experiment for predicting carbon dioxide, latent heat, sensible heat and net radiation fluxes across different ecosystem types with 11 machine learning (ML) methods from four different classes (kernel methods, neural networks, tree methods, and regression splines). We applied two complementary setups: (1) 8-day average fluxes based on remotely sensed data and (2) daily mean fluxes based on meteorological data and a mean seasonal cycle of remotely sensed variables. The patterns of predictions from different ML and experimental setups were highly consistent. There were systematic differences in performance among the fluxes, with the following ascending order: net ecosystem exchange (R20.6), gross primary production (R2>0.7), latent heat (R2>0.7), sensible heat (R2>0.7), and net radiation (R2>0.8). The ML methods predicted the across-site variability and the mean seasonal cycle of the observed fluxes very well (R2>0.7), while the 8-day deviations from the mean seasonal cycle were not well predicted (R2<0.5). Fluxes were better predicted at forested and temperate climate sites than at sites in extreme climates or less represented by training data (e.g., the tropics). The evaluated large ensemble of ML-based models will be the basis of new global flux productsISSN:1810-6277ISSN:1810-628

Predicting carbon dioxide and energy fluxes across global FLUXNET sites with regression algorithms

Abstract. Spatio-temporal fields of land–atmosphere fluxes derived from data-driven models can complement simulations by process-based land surface models. While a number of strategies for empirical models with eddy-covariance flux data have been applied, a systematic intercomparison of these methods has been missing so far. In this study, we performed a cross-validation experiment for predicting carbon dioxide, latent heat, sensible heat and net radiation fluxes across different ecosystem types with 11 machine learning (ML) methods from four different classes (kernel methods, neural networks, tree methods, and regression splines). We applied two complementary setups: (1) 8-day average fluxes based on remotely sensed data and (2) daily mean fluxes based on meteorological data and a mean seasonal cycle of remotely sensed variables. The patterns of predictions from different ML and experimental setups were highly consistent. There were systematic differences in performance among the fluxes, with the following ascending order: net ecosystem exchange (R2 0.6), gross primary production (R2> 0.7), latent heat (R2 > 0.7), sensible heat (R2 > 0.7), and net radiation (R2 > 0.8). The ML methods predicted the across-site variability and the mean seasonal cycle of the observed fluxes very well (R2 > 0.7), while the 8-day deviations from the mean seasonal cycle were not well predicted (R2 < 0.5). Fluxes were better predicted at forested and temperate climate sites than at sites in extreme climates or less represented by training data (e.g., the tropics). The evaluated large ensemble of ML-based models will be the basis of new global flux products